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ABSTRACT 

In redesigning the data program of the National 
Center for Education Statistics (NCES), there is a need for standard 
definitions, constant vigilance against redundancy and excessive data 
collection, audits and verification on local school data and 
continued attention to equity results as well as excellence and 
achievement. Major test scores, such as those obtained from the 
American College Testing Program and the Scholastic Aptitude Test 
should not be used to compare and contrast state achievement, since 
these tests were not designed for that purpose. The limits of the 
test instruments should appear in publications dealing with results 
of such tests. The following principles are suggested to guide NCES 
in data collection activity: (1) the data should be useful to 
federal, state and local policymakers or decision-makers; (2) the 
amount of data and number indicators should be limited to that which 
can be stored and analyzed within three months and reported to 
policymakers within the year; and (3) data should be drawn from 
sample rather than total populations. The National Assessment of 
Educational Progress is federally funded, and one option for 
including state assessments is to add funds to the contract. Two of 
the major concerns would be: (1) the cost of the expanded survey; and 
(2) timeliness of the reports. (JAZ) 
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The National Center for Education Statistics plays an important role for the federal 
government — collecting statistics useful for local, state and federal decision- 
making about education. I am pleased to join the discussion of how a redesign of 
NCES' data program will enhance opportunities to collect productive information 
and, further, promote innovative strategies for distribution, use and analysis. As my 
comments will illustrate, I support a simplified sampling system of educational 
attainment. The U.S. needs standard definitions, constant vigilance against 
redundancy and excessive data collection, audits and verification on local school 
data, and continued attention to equity results as well as excellence and 
achievement. 

During the 1960*s, I was one of several Harvard University professors of educational 
administration objecting to the position taken by the American Association of 
School Administrators to oppose national assessment and any suggestion of testing 
or federal action to find out what children had learned. After some discussion, 
AASA agreed to a compromise whereby a neutral state-oriented group, the 
Education Commission of the States, would conduct the national assessment of 
educational policy. Neither state nor local school districts would be identified in 
any report. 

Less than twenty years later, the governors and legislatures of most states insist on 
knowing how well the educatio'ial system is performing with results available by 
school district and often by school building. 



As the education officer in one Governor's cabinet and the chief state school officer 
in another state, I grew very dissatisfied with existing national education data and 
data collection formats. I remain very critical of the use by top U.S. officials — 
including the President and Secretary — of SAT and ACT results which cannot and 
should not be used to evaluate state achievement when the tests were intended only 
to predict college success for individuals. The SAT scores have been made the Dow 
Jones indicator of educational achievement. This is the wrong scoreboard and in- 
appropriate information to use to compare and contrast states. 

Fortunately, the Department of Education has begun to search for other indicators- 
the NAEP, as well as longitudinal studies on high school graduates, and other state 
level data. This is a positive trend. 

The Growing Demand for Educational Indicators 

For the last twenty-five years, the American public supported increased public 
expenditures for more comprehensive educational and health services — especially 
for those who lacked access to quality schools, clinics and hospitals. 

Neither educators nor medical administrators, however, felt obligated to compile or 
release to the public any systematic data about the accomplishments and failures 
of their institutions. Now the public wants to know, employers need to know, and 
government is under an obligation to collect and present these data in under- 
standable and responsible formats. 

Recently, the U.S. Department of Health and Human Services National Health 
Council voted to require every hospital to produce a profile of statistics which 
would be available to Medicare patients and the general public. Among the more 
compelling statistics would be the number of persons who die each year in each 
department of the hospital and from what ailment. Information on costs, admissions 
practices, length of stay, quality of staff, and a review of facilities are indeed vital 
statistics for hospital service consumers. Information on deaths per doctor will not 
be available due in part to very strong objections from the American Medical 
Association. Taxpayers and people seeking medical services will have considerable 
information due to this "truth in healing" policy and the help of professional review 
organizations which will monitor the hospitah. 
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Concurrently, an industrial location advisory service called FANTUS regularly 
informs companies looking for a new facility about the quality of state and local 
educational systems. An employer must know the literacy level of the local 
workforce from which new workers will be recruited. Also, employees with families 
traiisferred to the new site want to know the answer to the queries, "How good are 
the schools?" and "Where are the best schools?" FANTUS collects and summarizes 
all the available data on school size, costs, achievement, and other potentially 
significant indicators of quality, including the numbers of graduates going to 
college. 

Thirty-six states have required local schools to systematically test school achieve- 
ment for various reasons including grade to grade promotion and graduation from 
high school. New York State requires a series of tests of pupil performance 
evaluations well in advance of the Regents exams for college-bound students. 

The Limits on SAT/ACT Data 

The President of the United States and NCES ought not to try to make SAT (or 
ACT) scores the basis for state-by-state comparisons for these reasons: 

1. The Scholastic Aptitude Test is not a test of commonly taught or needed 
skills. The verbal test is a cluster of reading comprehension paragraphs 
(connmonly required skill) and a test of abilities to recognize antonyms and 
find analogies (which are much less commonly used in classrooms, in writing, 
or in life). The SAT scores, when compiled with high school grades and rank 
in class, do appear to predict success in college courses Freshman year. They 
do not purport to measure achievement in a variety of skills or subjects. 

2. The SAT yields only verbal and math scores. It does not, even in the verbal 
segnnent, measure writing, speaking or listening skills. The math is somewhat 
more comprehensive but there are separate College Board achievement tests 
in algebra, geometry, calculus, and computer sciences beyond wha^ the SAT 
test tries to measure. In addition, the SAT does not evalune aptitude in 
science, foreign language, history, health and other important subjects or skill 
clusters. 
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3. A different percentage of high school students take the test in each state. For 
example, two out of three students in Connecticut (63 percent) take the SAT 
exams; only one in three Texas students (33 percent) take the test. These 
variations make comparisons, let alone rankings of the state or conclusions 
about quality, impossible. The tests were designed to make judgments about 
each student, not their local schools and certainly not state school systems. 

^. State SAT scores, which are now ranked, ignore the demographic composition 
of each state. For example. New Hampshire, with high student scores on the 
SAT's, has one of the lowest number of minority students in the nation. The 
1.3 percent minority are mainly associated with the air base at Portsmouth 
where the median education level for minority parents in the service is 
actually greater than that of the white adult population of New Hampshire. 
Also, the percentage of handicapped students of New Hampshire is only three- 
fifths that of Massachusetts or New York. New Hampshire has no large cities, 
few minorities, and fewer than average handicapped. This may explain why, 
despite higher than average teacher -pupil ratios and low state support for 
schools. New Hampshire schools and academies produce comparatively high 
scores. State SAT scores for New Hampshire include not only local public high 
schools but large national college prep schools such as St. Paul's School 
(Concord) and Phillips Exeter Academy, almost all of whose students take and 
often score very high on the SAT test. 

The SAT was never designed to test statewide goals or provide state-by-state data. 
What is unacceptable is for the President of the United States to set state target 
scores on either the ACT or SAT aptitude tests since they are not achievement 
tests at all but specialized problem-solving measures useful for college admissions 
decisions. 

The "Performance Outcome" section of the State Education Statistics (January. 
1985) prepared by the Department of Education planning and evaluation service 
should be discontinued. Let the College Board (and ACT) release trend data as they 
do each September with the full notes about cautious interpretation. Cancel the 
publication of a simplistic Department of Education wall chart that does not suggest 
the limits of the test instrument, and which invites spurious comparison of unlike 
scores by the states. This chart serves poorly both college admission and national 
assessment movement. 
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What Should NCES Collect and Report? 



NCES wisely has begun the search for alternative indicators, has asked dozens of 
researchers and organizations for advice, and issued contracts to at least one 
university evaluation center. This extensive consultation is prudent and the ideas 
produced should be useful. 

I would like to suggest certain principles to guide data collection activity: 

1. The data should be useful to federal, strte and local policymakers or 
decision-makers. Much of educational research and assessment does not 
immediately suggest or lead to the development of a course of action. 
However, data on dropouts, bilingual education graduates, college-bound or 
job-bound students by occupation is very useful. 

2. The amount of data and number indicators should be limited to that which 
can be stored and analyzed within three months and reported to policy- 
makers within the year. Local and state school systems report much data 
to the federal government each year already that is stored but not 
summarized or used for other than formal report compliance purposes. 

3. Data should be drawn from sample rather than total populations. The 
information ordinarily will be just as useful and the cost of data collection, 
especially to local educators, will be dramatically reduced. 

Other analysts/contributors of advice to NIE will explain how the state of testing 
can now produce unlimited analyses of pupil performance on a thousand measures. 
But these important questions should be asked: Who needs it? Who will use it? 
For what purposes? These are deliberately hard questions, and they should be 
raised repeatedly about the entire program of data collection. 

The state profiles and "National Report Card" should reflect a consensus among 
state level educators as well as educational philosophers, psychologists and 
psychometricians about what is worth knowing about the schools. State agency 
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data collectors already have a strong sense of what information now is not used, 
or used very little or erroneously. Heed their advice. They, each and all, collect 
many reams of reports and tapes from local schools; they know the costs and can 
suggest which, if any, are beneficial. 

What would be useful for policymakers to know is student achievement in grades 
8, and 11 or 12 in these subject skills: 

Reading 

Writing - not proofreading but a composition test 
Computation 

Computer Skills - from keyboard to programming 

Listening 

Foreign Language 

Physical Education and Fitness 

Music 

Art appreciation 
Personal health and safety 
History and government 
Citizenship 

Also, it is important to know these indicators on a state by state basis: 

College (or postsecondary) intentions, and actual enrollment 
3ob placement, including Military 

Dropout rates - by age and grade (with a common definition) 
Handicapped student enrollment 

Bilingual student enrollment, and length of time in a program. 

The Secretary's report on excellence, A Nation At Risk , emphasized the "new 
basics" which were really the traditional academic subjects of reading, writing, 
mathematics, science, and only one "new" secondary school subject — computer 
studies. 
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The Secretary's recommendations on the curriculum were too limited. For 
several decades, the Council on Bask Education has included art, music and 
foreign langi:ages (all very traditional subjects associated with the cultivated, 
civilized, well-educated person) in their definition of basic history. NAEP 
included these subjects in the early assessments. The College Board and the New 
York Board of Regents provides achievement tests in those areas. So should any 
rigorous, systematic, national evaluation of education in the United States. 

One caveat about computers. Technology may be a more appropriate topic. 
Ernest Boyer, in his volume High School , points out the limits to teaching skills 
or computer languages that may be obsolete in five years. More so, he suggests 
that the study of machines, of systems, of the history and limits of technology 
is of more enduring importance. If so, work might properly begin on the outline 
of an evaluation strategy for the 1990's in a world of fibre optics, laser 
technology, artificial intelligence, and genetic engineering in which computers 
play a major role but not the only one worthy of systematic tudy. 

Finally, NAEP in the early yeari, measured the educational attainment of young 
adults (early 1920's). Since some youth complete high school at night or during 
their military service, these data are part of the full picture and should be once 
again collected and analyzed. 

Relevance, Quality and Utility of Data Collection Activities 

Usually, the federal government collects data by preparing survey instruments whfch 
are screened by one or more committees and then sent out to local schools, often 
by a state education agency whose staff helps to administer and monitor federal 
education programs. This is typically the way information is collected on vocational 
education, handicapped education programs. Chapter One basic skill programs, food 
and nutrition programs, and other services where the state and local responsibilities 
are shared. 

On several occasions, the task of data collection and analysis has been assigned to 
a contractor or center — as illustrated by the use, first, of the Education 
Commission of the States, and now. Educational Testing Service for the National 
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Assersment of "educational Progress. Also, the analysis of graduating seniors and 
their subsequent decisions was contracted to the National Opinion Research Center. 
These are, for the most part, appropriate alternatives to surveying state and local 
school systems directly for data on educational achievement and family decisions. 

The direct burden on local school systems to produce reports is much heavier than 
either researchers or federal program administrators realize. The U.S. has 
thousands of school districts with a few schools and a very small central office 
consisting of a superintendent and secretary, possibly a business manager. Weeks 
of work go into the preparation of periodic reports. 

As a chief state school officer, I began to realize that local school officials tended 
to blame the state since the survey forms and envelopes indicated the forms, with 
few exceptions, must be returned to the state education agency. This makes sense 
because each state needs the information, and states, not the federal government, 
have the constitutional requirement to keep track of local school resources and 
activities. But, as I argued in an article entitled "The Federal Takeover of 
Education," the federal government puts up 5-10 percent of the funds and accounts 
for 50 percent of the data requests of local schools. Data on handicapped children 
and programs on vocational education are among the most complex, voluminous and 
time-consuming surveys to complete. 

Since leaving the state education office, I have learned that local educators on 
occasion will meet a deadline by estimating the number of students served by a 
program. Rarely would anyone deliberately falsify statistics, even if the flow of 
dollars depended on a certain number. But the time and level of detail is difficult 
for small and medium-sized school districts and cumbersome for the large districts. 

Other issues of technical quality relate to definitions. Student dropout rates are 
defined differently by the several states. It cannot be assumed that the reported 
numbers carry the same meaning from state to state. A technical task force of 
federal and state educators should compile all of the definitions, point out the 
contradictions and anomalies, and propose a consensus position for the state boards, 
chief state school officers and NCES to adopt* 
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Also, the federal government must convene a group which would standardize the 
definition of handicapped terms. At present, the same words carry different 
meanings in these agencies: Office of Civil Rights, Vocational Education, NCES, 
Handicapped Education, Health and Human Services, and Bilingual Education (which 
has a legitimate need to know how many students are both bilingual and 
handicapped). A few states, such as Massachusetts, have legislated the abolition of 
handicapped labels which tend to stigmatize, denigrate and to reduce teacher 
expectations. 

However, certain federal bureaus and offices use outmoded or differing classifi- 
cation terms which make tabulation, comparison, and analysis quite difficult. 

The issues summarized above can be addressed by these approaches: 

1. Use of large and reputable data collection and analysis services for ':omplex, 
longitudinal and analytical work on achievement and effects of educators; 

2. Periodic verification of local and state data as reported on survey forms; 

3. Annual reviews of data reduction options and possible redundancies; 

4. A task force to review definitions of educational conditions, e.g., dropouts and 
handicapped; 

5. An effort to obtain a uniform federal definition of handicapped services and 
programs. 

Data Series Important to Administrators 

In my tenure as a state superintendent, I needed to be informed about pupil 
enrollment trends, school closings, consolidations, and emerging shortages of 
teachers and, eventually, of facilities. 

The data series on enrollments, school completions, percent in non-public school, 
length of school term, transportation trends, average attendance - these are bread 
and butter statistics needed to confirm or confront the conventional wisdom about 
school needs and trends. 
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For example, legislators suggest that if school enrollment is declining no new funds 
are needed. Yet, ali over America, class sizes are dropping and costs of education 
rising. These are important trends for education advocates and legislative analysts 
to consider. 

Data on race and linguistic background are important to administrators of bilingual 
programs and of intergroup or intercultural education. 

At the college level, the information on earned degrees, placements and salaries are 
important to institutional planners and statewide coordinators of higher education 
programs. 

Information on tuition, fees, and scholarship awards are much too skimpy given the 
importance of federal and state grant and loan programs. The Congress in 1980 
found great gaps in knowledge about these higher education aid programs (which 
consume $7 billion dollars or almost half of the federal education budget and some 
funds from other agencies and departments). Subsequently, Congress created a 
National Commission on Student Financial Assistance to conduct studies and collect 
data which NIE/NCES might have reported. Data on grant recipients, their income 
levels, their race and sex, their completion rates, default rates - all of these are 
important to Congress and to the states. NCES can begin by reviewing the 
questions assigned to the National Commission and the reports filed by Commission 
staff and consultants, many of which should be collected at least on a biennial basis. 

On the whole, this commentator believes that the NCES data series on finance and 
administration fills important information gaps about the resources made available 
to education and educators. 

Other Measures of Educational Productivity 

What data should be collected other than achievement data, test scores, the 
percentage passing courses or graduating from high school or college? 

Considerations for the 1990's include: 

The number of youth who participated in community service programs, required 
or voluntary, should be tabulated. The Carnegie Foundation for the Advance- 
ment of Teaching has advanced this proposal which some high schools have 




adopted. This 120 hour requirement becomes a new "Carnegie Unit" and since 
NCES maintains the scorecard, it is important to add this tally of community 
service/ citizenship education requirements. 

2. The amount of extra or co-curricular activities offered by U.S. high schools and 
the rate of student p'^rticipation. For fifty years the textbooks on school 
leadership have agreed that students learn t .m student council service, from 
debate club, band or chorus, basketball and soccer, future teachers or farmers, 
and other school activities 

3. The numbers of handicapped students who have been served in programs (such as 
those funded by 9^-1^2) and who have: 

a. been mainstreamed, sent to less restrictive alternatives, 

b. graduated into either vocational or college preparatory programs, 

c. become gainfully employed or enrolled in college. 

Now, one way to accomplish this is to engage in annual surveys. The other technique, 
potentially even more useful, is the periodic study of "high school and beyond," the 
systematic sampling of school graduates and t*ieir subsequent decisions about careers 
and continuing education. 

The same approach pertains to bilingual education where federal and state policy- 
makers need to know the answers to these questions: 

a. How many bilingual students remain in school and graduate? 

b. How many graduate from transitional bilingual education classes or 
programs? 

c. What is the level of linguistic competence, both in English and in 
another language, of those who participate in bilingual programs for 
one school year or more? 
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Data Collection Mrategies 

A strong argument has been advanced for broadening the role of the National As- 
sessment of Educational Progress to include state assessments as well. NA£P for 
several years has offered such a service to state education agencies. Several states, 
Including Minnesota and Connecticut, have made extensive use of the data col- 
lection format and many other states have obtained test items or subsections of the 
NAF.P instruments. 

NAEP is federally funded, and one option is to add funds to the contract to enable 
all states to be induced. Two of the major concerns will be: 

!. The cost of this expanded survey, especially if NAEP requires the hiring of 
local test administrators as is the current practice, and 

2. Timeliness of the reports, especially in more complicated assessments such as 
on writing skills. 

The issues of cost and timeliness will be important ones to address. If assessment 
is too labor-intensive and costly, then inevitably some components important to 
civilization such as art and music or citizenship education will be eliminated. 

Also, a report is useful to decision-makers if results are available within six to ten 
weeks — such as the College Board and ETS can provide to college admissions 
officials. According to some long-term participants in NCES and state education 
decision-making, the NAEP state survey data can take a year or more to analyze 
and report. A more timely analysis would presumably cost more money. 

Does NAEP need to cost more money to obtain state results? Three tasks must be 
undertaken: 

1. Legislators and chief state school officers should comment on the usefulness 
of the information produced by NAEP. Which findings or results are merely 
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interesting, and which provide data -nd analyses on which local and state 
decision-makers can act? 

2. State education agency staff should be asked whether NAEP should hire and 
assign test administrators for each local assessment site. In carrying out an 
assessment (an inventory of their educational progress)^ Illinois found that the 
expensive practice of assigning an assessment administrator was unnecessary. 
Illinois assessment results without the local assessment administrators were 
not significantly different. 

3. The speed of analysis and reporting must be timed to a decision-making cycle, 
to the planning/budgeting cycles of states (which vary) or to either a federal 
reauthorization or budget. Data must be current — which is one reason why 
the annual SAT score reports attract so much attention. 

If not NAEP, there remains the alternative of asking the larger state agencies to 
agree on common achievement data collection or at least a common core of testing 
and evaluation ^tivity. The Council of Chief State School Officers has begun this 
work and NCES/Department of Education should strongly support this effort. It is 
quite possible that assessment activity by a coalition of state educational agencies 
can be more cost-effective, more useful and more comprehensive than NAEP at 
present. 

This argues against an early decision (1986) on a single ten- year format for state- 
by-state assessments. The Department has already made serious errors in publishing 
state SAT and ACT scores. The Department should consult with state-based groups, 
especially the legislators and chiefs and state boards of education. States do 
respond to incentives, to capacity-building grants, and to cost-sharing programs. 
The state appetite for assessment data has grown enormously in the 1980*s, well 
ahead of any scientific consensus on how to evaluate education sensitively and 
thoroughly. This is an example of how federalism can work, a sharing of state and 
national resources. 

What seems highly desirable is an agreement to use the energy and commitment of 
the Council of Chief State School Officers to agree on common indicators of 
educational progress. Such a consultation should include the tests of comprehen- 
siveness, cost-effectiveness, and usefulness to state as well as federal policymakers. 
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